feat(house): 扩充看房数据并增强服务器日志功能
Some checks failed
Build and Deploy (service.xpcool.com) / build-and-deploy (push) Failing after 35s

- 新增新房预售证表 house_presale 存储预售许可信息
- 为 house_community 表添加 avg_price 字段存储小区参考均价
- 增强服务器操作审计日志功能,新增管理员账号、IP归属地、错误信息、UA等字段
- 实现纯Go版IP归属地离线解析器,无外部依赖,支持二分查找
- 优化看房列表查询逻辑,修复Fields设置位置导致的SQL语法错误
- 集成招聘模块,添加独立数据库配置和Bark推送服务支持
- 重构日志查询接口,支持多维度筛选和综合分页列表展示
- 更新DAO实体结构同步数据库表结构调整
This commit is contained in:
夏犀麟 2026-08-27 01:47:32 +08:00
parent 856d62f9c7
commit ea50261cb8
43 changed files with 1712 additions and 64 deletions

View File

@ -1,6 +1,10 @@
# service.xpcool.com 变更记录
> 倒序最新在上格式YYYY-MM-DD | 类型 | 摘要
2026-08-27 | CHG | 看房数据扩充013_house_presale.sql 建预售证表presale_no 唯一house_community avg_price 字段010 SQL 同步 + gen dao 更新 entity/do AvgPricego build 通过
2026-08-27 | CHG | 服务器日志操作审计 admin_operation_log全面增强为综合分页列表 新增字段 admin_username(冗余账号)ip_location(IP归属地)error_message(失败原因)user_agent 自建纯 Go xdb v4 离线解析器 internal/library/iplocgo:embed 嵌入 ip2region.xdb无外部依赖单次内存二分审计写入时解析账号+归属地记录 UA 与错误摘要 查询筛选覆盖时间范围/账号/IP/归属地/HTTP方法/结果(成功2xx·失败非2xx)/关键字(权限码·路径·IP)/耗时上下限/排序(时间·耗时) 接口 POST /api/service/admin/system/logbody 入参遵守全 POST 无参约定dto.LogQuery/LogItem 同步 SQL001_core.sql 建表补列 + 新增 010_server_log_enhance.sql 增量迁移须手动导库DB 结构变更不自动 DDLgo build 通过iploc 单元冒烟通过
2026-08-27 | FIX | 看房中心接口异常修复根因=后端进程是旧代码recruitment 模块中途编译错误致项目编译不过后端停在旧进程未加载 house 路由重启后端加载新代码附带修复 house listing/transaction List 方法 Fields Count 前设置导致 COUNT(t.*,c.name) SQL 语法错误Fields 移到 Scan 7 个看房接口全部 code 0 验证通过
2026-08-27 | CFG | 新增 013_workbench_menu.sql工作台顶层菜单id=9, parent_id=0, path=/workbench, component=dashboard/workbench/index, permission=workbench, sort=0 置顶+ 超管 role_id=1 绑定前端工作台页面复用既有列表接口聚合无新增 Go 接口
2026-08-27 | CHG | 看房成交模块api/house/transaction + service/house/transaction + controller/house/transaction 实现成交记录 list 接口/api/service/admin/house/transaction/list POST 分页关联小区名按成交日期倒序dto HouseTransactionVOcmd.go 注册012_house_transaction_menu.sql 权限种子house:data:transaction id=941 挂数据明细 94go build 通过
2026-08-26 | CFG | 新增 Gitea act_runner 自动部署.gitea/workflows/deploy.ymlpush main/dev 下载 Go1.23 工具链 npmmirror + GOPROXY goproxy.cn CGO=0 linux/amd64 交叉编译 组装 main+manifest/config(config.yaml+config.prod.yaml)+resource+deploy/Dockerfile docker build 重建容器 127.0.0.1:10100xpcool-netGF_GCFG_ENV=prodDB_DSN/JWT_SECRET Gitea Actions secrets curl /api.json 冒烟deploy/Dockerfile 仓库内维护DB 结构变更不自动 DDL需手动导 manifest/sql
2026-08-26 | FIX | 本地登录 no rows 修复根因=本地库 service_xpcool_com admin_user 空表本地/服务器库两套此前仅改服务器 super_admin 角色 + 导入 009_admin_account_v2.sql --default-character-set=utf8mb4 否则中文 Data too long 本地 xxcool/xxCool@2026 登录成功subject=3同时服务器废弃 webhook 方案已删除deploy-webhook.service/webhook.py/secrets.env/deploy.sh自动部署改走 Gitea act_runner + .gitea/workflows仓库内尚未配置待补

View File

@ -0,0 +1,36 @@
// Package house_presale 定义新房预售证查询接口,路由前缀 /house。
package house_presale
import "github.com/gogf/gf/v2/frame/g"
// PresaleItem 一条新房预售许可证。
type PresaleItem struct {
Id uint64 `json:"id"`
PresaleNo string `json:"presaleNo"`
CommunityName string `json:"communityName"`
Developer string `json:"developer"`
Region string `json:"region"`
Address string `json:"address"`
BuildingNo string `json:"buildingNo"`
HouseCount int `json:"houseCount"`
Area float64 `json:"area"`
Purpose string `json:"purpose"`
IssueDate string `json:"issueDate"`
Source string `json:"source"`
}
// PresaleListReq 分页查询新房预售证。
type PresaleListReq struct {
g.Meta `path:"/house/presale/list" method:"post" tags:"Admin/House/Presale" summary:"新房预售证列表"`
Page int `json:"page" d:"1" v:"min:1"`
Size int `json:"size" d:"10" v:"min:1|max:100"`
Region string `json:"region"`
Purpose string `json:"purpose"` // 规划用途(住宅/商业)
Keyword string `json:"keyword"` // 匹配楼盘名/开发商/预售证号
}
// PresaleListRes 是 PresaleListReq 的响应。
type PresaleListRes struct {
List []*PresaleItem `json:"list"`
Total int `json:"total"`
}

2
go.sum
View File

@ -33,6 +33,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lionsoul2014/ip2region v3.17.0+incompatible h1:91PbwotwoEMPV6nfAALL+WTmYHmTuRNFgJX6T8uPUNY=
github.com/lionsoul2014/ip2region v3.17.0+incompatible/go.mod h1:+ZBN7PBoh5gG6/y0ZQ85vJDBe21WnfbRrQQwTfliJJI=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=

View File

@ -6,7 +6,7 @@ gfcli:
dao:
- link: "mysql:root:root123@tcp(127.0.0.1:3306)/service_xpcool_com"
descriptionTag: true
tables: "house_community,house_building,house_listing,house_price_snapshot,house_transaction,house_facility,house_community_facility,house_school_district,house_preference"
tables: "house_community,house_building,house_listing,house_price_snapshot,house_transaction,house_facility,house_community_facility,house_school_district,house_preference,house_presale"
docker:
build: "-a amd64 -s linux -p temp -ew"

View File

@ -12,6 +12,7 @@ import (
adminctl "service.xpcool.com/internal/controller/admin"
housectl "service.xpcool.com/internal/controller/house"
openctl "service.xpcool.com/internal/controller/open"
recruitmentctl "service.xpcool.com/internal/controller/recruitment"
userctl "service.xpcool.com/internal/controller/user"
"service.xpcool.com/internal/library/jwt"
"service.xpcool.com/internal/middleware"
@ -28,6 +29,8 @@ import (
housedashboard "service.xpcool.com/internal/service/house/dashboard"
houselisting "service.xpcool.com/internal/service/house/listing"
housetransaction "service.xpcool.com/internal/service/house/transaction"
housepresale "service.xpcool.com/internal/service/house/presale"
recruitmentsvc "service.xpcool.com/internal/service/recruitment"
)
// injectEnv 手动把关键环境变量写入配置系统。
@ -44,6 +47,19 @@ func injectEnv(ctx context.Context) {
if v := genv.Get("JWT_SECRET"); !v.IsEmpty() {
_ = adapter.Set("jwt.secret", v.String())
}
// 招聘模块独立数据库与 Bark 推送配置(自建 Bark 服务)。
if v := genv.Get("RECRUITMENT_DB_DSN"); !v.IsEmpty() {
_ = adapter.Set("database.recruitment.link", v.String())
}
if v := genv.Get("BARK_BASE_URL"); !v.IsEmpty() {
_ = adapter.Set("bark.baseUrl", v.String())
}
if v := genv.Get("BARK_DEVICE_KEY"); !v.IsEmpty() {
_ = adapter.Set("bark.deviceKey", v.String())
}
if v := genv.Get("BARK_PUSH_TIME"); !v.IsEmpty() {
_ = adapter.Set("bark.pushTime", v.String())
}
}
var (
@ -68,6 +84,8 @@ var (
houselisting.RegisterListing(houselisting.NewListing())
housedashboard.RegisterDashboard(housedashboard.NewDashboard())
housetransaction.RegisterTransaction(housetransaction.NewTransaction())
housepresale.RegisterPresale(housepresale.NewPresale())
recruitmentsvc.RegisterRecruitment(recruitmentsvc.New())
s.Group("/api/service/open", func(group *ghttp.RouterGroup) {
group.Middleware(middleware.Recover, middleware.CORS, ghttp.MiddlewareHandlerResponse)
group.Bind(openctl.New()) // Open tools API for frontends, no auth required.
@ -88,13 +106,16 @@ var (
group.Group("/", func(protected *ghttp.RouterGroup) {
// 受权限保护端点RBAC 管理、日志等。
// 权限由后端按「方法+路径」自动匹配,无需前端传 X-Permission。
protected.Middleware(middleware.AdminAuth(tokens, adminauth.AdminAuth().PermissionForPath, adminauth.AdminAuth().HasPermission, func(ctx context.Context, id uint64, permission, method, path, ip, param string, duration, status int) {
adminaudit.AdminAudit().Record(ctx, adminaudit.AuditEvent{AdminID: id, Permission: permission, Method: method, Path: path, IP: ip, Param: param, DurationMS: duration, StatusCode: status})
protected.Middleware(middleware.AdminAuth(tokens, adminauth.AdminAuth().PermissionForPath, adminauth.AdminAuth().HasPermission, func(ctx context.Context, id uint64, permission, method, path, ip, param string, duration, status int, userAgent, errorMessage string) {
adminaudit.AdminAudit().Record(ctx, adminaudit.AuditEvent{AdminID: id, Permission: permission, Method: method, Path: path, IP: ip, Param: param, DurationMS: duration, StatusCode: status, ErrorMessage: errorMessage, UserAgent: userAgent})
}))
protected.Bind(adminctl.New())
protected.Bind(housectl.New())
protected.Bind(recruitmentctl.New())
})
})
// 启动招聘模块定时任务(每日增量抓取 + 早报推送)。
recruitmentsvc.StartScheduler(ctx)
s.Run()
return nil
},

View File

@ -34,16 +34,24 @@ func (c *Controller) LogTail(ctx context.Context, req *logv1.LogTailReq) (res *l
// LogList 分页查询 admin 系统日志(操作审计记录)。
func (c *Controller) LogList(ctx context.Context, req *systemlogv1.LogListReq) (res *systemlogv1.LogListRes, err error) {
items, total, err := adminlog.AdminAudit().List(ctx, dto.LogQuery{Page: req.Page, Size: req.Size, AdminID: req.AdminID, Permission: req.Keyword})
items, total, err := adminlog.AdminAudit().List(ctx, dto.LogQuery{
Page: req.Page, Size: req.Size, AdminID: req.AdminID, Username: req.Username,
IP: req.IP, IpLocation: req.IpLocation, Method: req.Method, Status: req.Status,
Keyword: req.Keyword, StartTime: req.StartTime, EndTime: req.EndTime,
MinDuration: req.MinDuration, MaxDuration: req.MaxDuration,
OrderBy: req.OrderBy, OrderDir: req.OrderDir,
})
if err != nil {
return nil, err
}
list := make([]*systemlogv1.LogItem, 0, len(items))
for _, it := range items {
list = append(list, &systemlogv1.LogItem{
Id: it.Id, AdminID: it.AdminID, Permission: it.Permission, Method: it.Method,
Path: it.Path, IP: it.IP, Param: it.Param, DurationMS: it.DurationMS,
StatusCode: it.StatusCode, CreatedAt: it.CreatedAt,
Id: it.Id, AdminID: it.AdminID, AdminUsername: it.AdminUsername,
Permission: it.Permission, Method: it.Method, Path: it.Path, IP: it.IP,
IpLocation: it.IpLocation, Param: it.Param, DurationMS: it.DurationMS,
StatusCode: it.StatusCode, ErrorMessage: it.ErrorMessage,
UserAgent: it.UserAgent, CreatedAt: it.CreatedAt,
})
}
return &systemlogv1.LogListRes{List: list, Total: total}, nil

View File

@ -0,0 +1,35 @@
package house
import (
"context"
presalev1 "service.xpcool.com/api/house/presale"
presale "service.xpcool.com/internal/service/house/presale"
)
// PresaleList 分页查询新房预售证。
func (c *Controller) PresaleList(ctx context.Context, req *presalev1.PresaleListReq) (res *presalev1.PresaleListRes, err error) {
list, total, err := presale.Presale().List(ctx, req.Page, req.Size, req.Region, req.Keyword, req.Purpose)
if err != nil {
return nil, err
}
out := make([]*presalev1.PresaleItem, 0, len(list))
for i := range list {
v := &list[i]
out = append(out, &presalev1.PresaleItem{
Id: v.Id,
PresaleNo: v.PresaleNo,
CommunityName: v.CommunityName,
Developer: v.Developer,
Region: v.Region,
Address: v.Address,
BuildingNo: v.BuildingNo,
HouseCount: v.HouseCount,
Area: v.Area,
Purpose: v.Purpose,
IssueDate: v.IssueDate,
Source: v.Source,
})
}
return &presalev1.PresaleListRes{List: out, Total: total}, nil
}

View File

@ -0,0 +1,22 @@
// =================================================================================
// This file is auto-generated by the GoFrame CLI tool. You may modify it as needed.
// =================================================================================
package dao
import (
"service.xpcool.com/internal/dao/internal"
)
// housePresaleDao is the data access object for the table house_presale.
// You can define custom methods on it to extend its functionality as needed.
type housePresaleDao struct {
*internal.HousePresaleDao
}
var (
// HousePresale is a globally accessible object for table house_presale operations.
HousePresale = housePresaleDao{internal.NewHousePresaleDao()}
)
// Add your custom methods and functionality below.

View File

@ -32,6 +32,7 @@ type HouseCommunityColumns struct {
Households string // 总户数
PlotRatio string // 容积率
GreenRate string // 绿化率
AvgPrice string // 小区参考均价(元/平米)
PropertyCompany string // 物业公司
PropertyFee string // 物业费(元/月/平米)
Developer string // 开发商
@ -54,6 +55,7 @@ var houseCommunityColumns = HouseCommunityColumns{
Households: "households",
PlotRatio: "plot_ratio",
GreenRate: "green_rate",
AvgPrice: "avg_price",
PropertyCompany: "property_company",
PropertyFee: "property_fee",
Developer: "developer",

View File

@ -0,0 +1,107 @@
// ==========================================================================
// Code generated and maintained by GoFrame CLI tool. DO NOT EDIT.
// ==========================================================================
package internal
import (
"context"
"github.com/gogf/gf/v2/database/gdb"
"github.com/gogf/gf/v2/frame/g"
)
// HousePresaleDao is the data access object for the table house_presale.
type HousePresaleDao struct {
table string // table is the underlying table name of the DAO.
group string // group is the database configuration group name of the current DAO.
columns HousePresaleColumns // columns contains all the column names of Table for convenient usage.
handlers []gdb.ModelHandler // handlers for customized model modification.
}
// HousePresaleColumns defines and stores column names for the table house_presale.
type HousePresaleColumns struct {
Id string //
PresaleNo string // 预售证号
CommunityName string // 楼盘/项目名
Developer string // 房地产开发企业
Region string // 区县
Address string // 建筑位置
BuildingNo string // 楼栋号
HouseCount string // 套数
Area string // 建筑面积(平米)
Purpose string // 规划用途(住宅/商业)
IssueDate string // 初始核发日期
Source string // 数据来源
CreatedAt string //
UpdatedAt string //
DeletedAt string //
}
// housePresaleColumns holds the columns for the table house_presale.
var housePresaleColumns = HousePresaleColumns{
Id: "id",
PresaleNo: "presale_no",
CommunityName: "community_name",
Developer: "developer",
Region: "region",
Address: "address",
BuildingNo: "building_no",
HouseCount: "house_count",
Area: "area",
Purpose: "purpose",
IssueDate: "issue_date",
Source: "source",
CreatedAt: "created_at",
UpdatedAt: "updated_at",
DeletedAt: "deleted_at",
}
// NewHousePresaleDao creates and returns a new DAO object for table data access.
func NewHousePresaleDao(handlers ...gdb.ModelHandler) *HousePresaleDao {
return &HousePresaleDao{
group: "default",
table: "house_presale",
columns: housePresaleColumns,
handlers: handlers,
}
}
// DB retrieves and returns the underlying raw database management object of the current DAO.
func (dao *HousePresaleDao) DB() gdb.DB {
return g.DB(dao.group)
}
// Table returns the table name of the current DAO.
func (dao *HousePresaleDao) Table() string {
return dao.table
}
// Columns returns all column names of the current DAO.
func (dao *HousePresaleDao) Columns() HousePresaleColumns {
return dao.columns
}
// Group returns the database configuration group name of the current DAO.
func (dao *HousePresaleDao) Group() string {
return dao.group
}
// Ctx creates and returns a Model for the current DAO. It automatically sets the context for the current operation.
func (dao *HousePresaleDao) Ctx(ctx context.Context) *gdb.Model {
model := dao.DB().Model(dao.table)
for _, handler := range dao.handlers {
model = handler(model)
}
return model.Safe().Ctx(ctx)
}
// Transaction wraps the transaction logic using function f.
// It rolls back the transaction and returns the error if function f returns a non-nil error.
// It commits the transaction and returns nil if function f returns nil.
//
// Note: Do not commit or roll back the transaction in function f,
// as it is automatically handled by this function.
func (dao *HousePresaleDao) Transaction(ctx context.Context, f func(ctx context.Context, tx gdb.TX) error) (err error) {
return dao.Ctx(ctx).Transaction(ctx, f)
}

Binary file not shown.

View File

@ -0,0 +1,99 @@
// Package iploc 提供离线 IP 归属地解析(省/市/运营商)。
// 底层数据使用 ip2region 的 xdb(v4) 数据库,通过 go:embed 直接打进二进制,
// 无需外部文件、无需联网,单次查询为内存二分查找,性能稳定。
package iploc
import (
_ "embed"
"encoding/binary"
"net"
"strings"
)
//go:embed ip2region.xdb
var xdbContent []byte
// xdb v4 文件结构常量(与官方 C binding 保持一致)。
const (
headerInfoLength = 256 // 头部信息长度
vectorIndexCols = 256 // 向量索引列数
vectorIndexSize = 8 // 每条向量索引 8 字节(起始 ptr + 结束 ptr
v4IndexSize = 14 // IPv4 段索引块4(IP 起) + 4(IP 止) + 2(数据长度) + 4(数据偏移)
)
// Locate 解析 IPv4 的归属地,返回形如 "广东 深圳 电信" 的可读字符串。
// 内部/无效/解析失败的 IP 返回空字符串(调用方据此决定展示策略)。
// 备注:当前仅支持 IPv4IPv6 直接返回空,避免误判。
func Locate(ipStr string) string {
ip := net.ParseIP(ipStr)
if ip == nil {
return ""
}
v4 := ip.To4()
if v4 == nil {
// 暂不支持 IPv6 归属地解析,返回空。
return ""
}
// v4 已是网络序 4 字节 [a,b,c,d],按大端整型比较即可与段索引块中的起止 IP 对齐。
ipUint := binary.BigEndian.Uint32(v4)
// 1) 由前两个字节定位向量索引,得到段索引区间 [sPtr, ePtr]。
il0 := int(v4[0])
il1 := int(v4[1])
idx := il0*vectorIndexCols*vectorIndexSize + il1*vectorIndexSize
vOff := headerInfoLength + idx
if vOff+8 > len(xdbContent) {
return ""
}
sPtr := binary.LittleEndian.Uint32(xdbContent[vOff : vOff+4])
ePtr := binary.LittleEndian.Uint32(xdbContent[vOff+4 : vOff+8])
if sPtr == 0 || ePtr == 0 {
// 该前缀段无数据,归属地未知。
return ""
}
// 2) 在段索引区间中二分查找命中 IP 的段(段索引块按起始 IP 有序)。
l, h := 0, int((ePtr-sPtr)/v4IndexSize)
dataPtr, dataLen := uint32(0), uint16(0)
for l <= h {
m := (l + h) >> 1
p := sPtr + uint32(m)*v4IndexSize
if int(p)+v4IndexSize > len(xdbContent) {
break
}
startIP := binary.BigEndian.Uint32(xdbContent[p : p+4])
endIP := binary.BigEndian.Uint32(xdbContent[p+4 : p+8])
switch {
case ipUint < startIP:
h = m - 1
case ipUint > endIP:
l = m + 1
default:
// 命中:数据长度在偏移 8小端 uint16数据偏移在偏移 10小端 uint32
dataLen = binary.LittleEndian.Uint16(xdbContent[p+8 : p+10])
dataPtr = binary.LittleEndian.Uint32(xdbContent[p+10 : p+14])
l, h = 0, -1 // 退出循环
}
}
if dataLen == 0 {
return ""
}
if int(dataPtr)+int(dataLen) > len(xdbContent) {
return ""
}
return cleanRegion(string(xdbContent[dataPtr : dataPtr+uint32(dataLen)]))
}
// cleanRegion 将 "国家|区域|省份|城市|运营商" 中的 0/空 段剔除,拼接为可读归属地。
func cleanRegion(raw string) string {
parts := strings.Split(raw, "|")
out := make([]string, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p == "" || p == "0" {
continue
}
out = append(out, p)
}
return strings.Join(out, " ")
}

View File

@ -0,0 +1,20 @@
package iploc
import "testing"
func TestLocateSmoke(t *testing.T) {
cases := []string{
"8.8.8.8", // Google 公共 DNS美国
"1.1.1.1", // Cloudflare
"114.114.114.114", // 国内公共 DNS
"180.76.76.76", // 百度
"218.4.116.1", // 江苏电信
"192.168.1.1", // 内网
"127.0.0.1", // 回环
"not-an-ip", // 非法
"::1", // IPv6暂不支持
}
for _, ip := range cases {
t.Logf("ip=%-18s -> %q", ip, Locate(ip))
}
}

View File

@ -2,6 +2,7 @@ package middleware
import (
"context"
"encoding/json"
"github.com/gogf/gf/v2/net/ghttp"
"service.xpcool.com/internal/consts"
"service.xpcool.com/internal/library/jwt"
@ -30,6 +31,25 @@ func auditParam(raw string) string {
}
return raw
}
// extractErrorMsg 当响应状态码 >= 400 时,从响应体 JSON 中提取 msg 作为失败原因摘要。
// 非失败场景返回空字符串,避免无意义写入。
func extractErrorMsg(r *ghttp.Request) string {
if r.Response.Status < 400 {
return ""
}
body := r.Response.BufferString()
if len(body) == 0 {
return ""
}
var out struct {
Msg string `json:"msg"`
}
if err := json.Unmarshal([]byte(body), &out); err == nil && out.Msg != "" {
return out.Msg
}
return string(body)
}
func UserAuth(s *jwt.Service) ghttp.HandlerFunc {
// 用户端仅接受 scope=user 的 access token。
return func(r *ghttp.Request) {
@ -55,7 +75,7 @@ func AdminAuthOnly(s *jwt.Service) ghttp.HandlerFunc {
r.Middleware.Next()
}
}
func AdminAuth(s *jwt.Service, permissionLookup func(context.Context, string, string) (string, error), permissionCheck func(context.Context, uint64, string) (bool, error), audit func(context.Context, uint64, string, string, string, string, string, int, int)) ghttp.HandlerFunc {
func AdminAuth(s *jwt.Service, permissionLookup func(context.Context, string, string) (string, error), permissionCheck func(context.Context, uint64, string) (bool, error), audit func(context.Context, uint64, string, string, string, string, string, int, int, string, string)) ghttp.HandlerFunc {
// 管理端接口鉴权:先解析 admin token再按「请求方法+路径」反查所需权限码
// admin_menu type=2 行的 path 映射),最后校验该管理员是否拥有该权限码。
// 未配置映射的接口一律拒绝,防止用任意已拥有权限码越权访问。
@ -77,7 +97,8 @@ func AdminAuth(s *jwt.Service, permissionLookup func(context.Context, string, st
return
}
defer func() {
audit(r.Context(), c.Subject, permission, r.Method, r.URL.Path, r.GetClientIp(), auditParam(r.GetBodyString()), int(time.Since(start).Milliseconds()), r.Response.Status)
// 失败时从响应体解析 msg 作为错误摘要UA 直接取请求头。
audit(r.Context(), c.Subject, permission, r.Method, r.URL.Path, r.GetClientIp(), auditParam(r.GetBodyString()), int(time.Since(start).Milliseconds()), r.Response.Status, r.Header.Get("User-Agent"), extractErrorMsg(r))
}()
r.SetCtxVar(AdminIDKey, c.Subject)
r.SetCtxVar(PermissionKey, permission)

View File

@ -14,13 +14,17 @@ type AdminOperationLog struct {
g.Meta `orm:"table:admin_operation_log, do:true"`
Id any //
AdminUserId any //
AdminUsername any // 管理员账号(冗余)
Permission any //
Method any //
Path any //
Ip any //
IpLocation any // IP 归属地
RequestParam any //
DurationMs any //
StatusCode any //
ErrorMessage any // 失败原因
UserAgent any // 浏览器 UA
CreatedAt *gtime.Time //
UpdatedAt *gtime.Time //
DeletedAt *gtime.Time //

View File

@ -22,6 +22,7 @@ type HouseCommunity struct {
Households any // 总户数
PlotRatio any // 容积率
GreenRate any // 绿化率
AvgPrice any // 小区参考均价(元/平米)
PropertyCompany any // 物业公司
PropertyFee any // 物业费(元/月/平米)
Developer any // 开发商

View File

@ -0,0 +1,29 @@
// =================================================================================
// Code generated and maintained by GoFrame CLI tool. DO NOT EDIT.
// =================================================================================
package do
import (
"github.com/gogf/gf/v2/frame/g"
)
// HousePresale is the golang structure of table house_presale for DAO operations like Where/Data.
type HousePresale struct {
g.Meta `orm:"table:house_presale, do:true"`
Id any //
PresaleNo any // 预售证号
CommunityName any // 楼盘/项目名
Developer any // 房地产开发企业
Region any // 区县
Address any // 建筑位置
BuildingNo any // 楼栋号
HouseCount any // 套数
Area any // 建筑面积(平米)
Purpose any // 规划用途(住宅/商业)
IssueDate any // 初始核发日期
Source any // 数据来源
CreatedAt any //
UpdatedAt any //
DeletedAt any //
}

View File

@ -79,6 +79,11 @@ type HouseTransactionVO struct {
CommunityName string `json:"communityName" orm:"community_name"`
}
// HousePresaleVO 新房预售证查询结果(表内已含楼盘名,无需关联)。
type HousePresaleVO struct {
entity.HousePresale
}
// DashboardOverview 看板统计概览。
type DashboardOverview struct {
CommunityCount int

View File

@ -9,25 +9,40 @@ type LogFile struct {
ModTime string
}
// LogQuery 管理员操作日志admin 系统日志)分页查询参数
// LogQuery 管理员操作日志admin 系统日志)分页查询参数覆盖时间、账号、IP、方法、结果、耗时、排序等维度
type LogQuery struct {
Page int
Size int
AdminID uint64 // 按管理员过滤
Permission string // 按权限码/路径关键字过滤
AdminID uint64 // 按管理员ID精确过滤
Username string // 按管理员账号模糊过滤(冗余字段,便于检索)
IP string // 按来源 IP 模糊过滤
IpLocation string // 按 IP 归属地模糊过滤
Method string // 按 HTTP 方法精确过滤GET/POST/PUT/DELETE...
Status int // 结果筛选0 全部 / 1 成功(2xx) / 2 失败(非2xx)
Keyword string // 关键字:匹配 permission / path / ip
StartTime string // 起始时间 created_at >=,格式 2006-01-02 15:04:05
EndTime string // 结束时间 created_at <=,格式 2006-01-02 15:04:05
MinDuration int // 耗时下限(ms)0 表示不限
MaxDuration int // 耗时上限(ms)0 表示不限
OrderBy string // 排序字段createdAt(默认) | durationMs
OrderDir string // 排序方向desc(默认) | asc
}
// LogItem 一条管理员操作日志记录。
type LogItem struct {
Id uint64
AdminID uint64
AdminUsername string
Permission string
Method string
Path string
IP string
IpLocation string
Param string
DurationMS uint
StatusCode int
ErrorMessage string
UserAgent string
CreatedAt string
}

View File

@ -119,7 +119,7 @@ type CrawlRunResult struct {
SourceId uint64
Fetched int
NewCount int
Updated int
UpdatedCount int
Err error
}

View File

@ -12,13 +12,17 @@ import (
type AdminOperationLog struct {
Id uint64 `json:"id" orm:"id" description:""` //
AdminUserId uint64 `json:"adminUserId" orm:"admin_user_id" description:""` //
AdminUsername string `json:"adminUsername" orm:"admin_username" description:"管理员账号(冗余)"` //
Permission string `json:"permission" orm:"permission" description:""` //
Method string `json:"method" orm:"method" description:""` //
Path string `json:"path" orm:"path" description:""` //
Ip string `json:"ip" orm:"ip" description:""` //
IpLocation string `json:"ipLocation" orm:"ip_location" description:"IP 归属地"` //
RequestParam string `json:"requestParam" orm:"request_param" description:""` //
DurationMs uint `json:"durationMs" orm:"duration_ms" description:""` //
StatusCode int `json:"statusCode" orm:"status_code" description:""` //
ErrorMessage string `json:"errorMessage" orm:"error_message" description:"失败原因"` //
UserAgent string `json:"userAgent" orm:"user_agent" description:"浏览器 UA"` //
CreatedAt *gtime.Time `json:"createdAt" orm:"created_at" description:""` //
UpdatedAt *gtime.Time `json:"updatedAt" orm:"updated_at" description:""` //
DeletedAt *gtime.Time `json:"deletedAt" orm:"deleted_at" description:""` //

View File

@ -17,6 +17,7 @@ type HouseCommunity struct {
Households int `json:"households" orm:"households" description:"总户数"` // 总户数
PlotRatio float64 `json:"plotRatio" orm:"plot_ratio" description:"容积率"` // 容积率
GreenRate float64 `json:"greenRate" orm:"green_rate" description:"绿化率"` // 绿化率
AvgPrice float64 `json:"avgPrice" orm:"avg_price" description:"小区参考均价(元/平米)"` // 小区参考均价(元/平米)
PropertyCompany string `json:"propertyCompany" orm:"property_company" description:"物业公司"` // 物业公司
PropertyFee float64 `json:"propertyFee" orm:"property_fee" description:"物业费(元/月/平米)"` // 物业费(元/月/平米)
Developer string `json:"developer" orm:"developer" description:"开发商"` // 开发商

View File

@ -0,0 +1,24 @@
// =================================================================================
// Code generated and maintained by GoFrame CLI tool. DO NOT EDIT.
// =================================================================================
package entity
// HousePresale is the golang structure for table house_presale.
type HousePresale struct {
Id uint64 `json:"id" orm:"id" description:""` //
PresaleNo string `json:"presaleNo" orm:"presale_no" description:"预售证号"` // 预售证号
CommunityName string `json:"communityName" orm:"community_name" description:"楼盘/项目名"` // 楼盘/项目名
Developer string `json:"developer" orm:"developer" description:"房地产开发企业"` // 房地产开发企业
Region string `json:"region" orm:"region" description:"区县"` // 区县
Address string `json:"address" orm:"address" description:"建筑位置"` // 建筑位置
BuildingNo string `json:"buildingNo" orm:"building_no" description:"楼栋号"` // 楼栋号
HouseCount int `json:"houseCount" orm:"house_count" description:"套数"` // 套数
Area float64 `json:"area" orm:"area" description:"建筑面积(平米)"` // 建筑面积(平米)
Purpose string `json:"purpose" orm:"purpose" description:"规划用途(住宅/商业)"` // 规划用途(住宅/商业)
IssueDate string `json:"issueDate" orm:"issue_date" description:"初始核发日期"` // 初始核发日期
Source string `json:"source" orm:"source" description:"数据来源"` // 数据来源
CreatedAt string `json:"createdAt" orm:"created_at" description:""` //
UpdatedAt string `json:"updatedAt" orm:"updated_at" description:""` //
DeletedAt string `json:"deletedAt" orm:"deleted_at" description:""` //
}

View File

@ -42,7 +42,6 @@ func RegisterListing(i IListing) { localListing = i }
// List 分页查询房源,关联小区名,支持多维筛选(管理列表与看板共用)。
func (s *listing) List(ctx context.Context, f dto.HouseListingFilter) ([]dto.HouseListingVO, int, error) {
m := dao.HouseListing.Ctx(ctx).As("l").LeftJoin("house_community c", "l.community_id=c.id")
m = m.Fields("l.*, c.name AS community_name")
if f.CommunityId > 0 {
m = m.Where("l.community_id", f.CommunityId)
}
@ -84,7 +83,7 @@ func (s *listing) List(ctx context.Context, f dto.HouseListingFilter) ([]dto.Hou
return nil, 0, gerror.Wrap(err, "count listing")
}
var list []dto.HouseListingVO
if err = m.Clone().Page(f.Page, f.Size).OrderDesc("l.id").Scan(&list); err != nil {
if err = m.Clone().Fields("l.*, c.name AS community_name").Page(f.Page, f.Size).OrderDesc("l.id").Scan(&list); err != nil {
return nil, 0, gerror.Wrap(err, "query listing list")
}
return list, total, nil

View File

@ -0,0 +1,57 @@
// Package house_presale 提供新房预售证领域服务。
package house_presale
import (
"context"
"github.com/gogf/gf/v2/errors/gerror"
"service.xpcool.com/internal/dao"
"service.xpcool.com/internal/model/dto"
)
// IPresale 新房预售证领域服务接口。
type IPresale interface {
List(context.Context, int, int, string, string, string) ([]dto.HousePresaleVO, int, error)
}
type presale struct{}
var localPresale IPresale
func NewPresale() IPresale { return &presale{} }
// Presale 返回已注册的预售证服务实现。
func Presale() IPresale {
if localPresale == nil {
panic("Presale implementation not registered")
}
return localPresale
}
// RegisterPresale 注册预售证服务实现。
func RegisterPresale(i IPresale) { localPresale = i }
// List 分页查询新房预售证,按 id 倒序(最新发证在前)。
func (s *presale) List(ctx context.Context, page, size int, region, keyword, purpose string) ([]dto.HousePresaleVO, int, error) {
m := dao.HousePresale.Ctx(ctx)
if region != "" {
m = m.Where("region", region)
}
if purpose != "" {
m = m.Where("purpose", purpose)
}
if keyword != "" {
m = m.Where("community_name LIKE ? OR developer LIKE ? OR presale_no LIKE ?",
"%"+keyword+"%", "%"+keyword+"%", "%"+keyword+"%")
}
total, err := m.Clone().Count()
if err != nil {
return nil, 0, gerror.Wrap(err, "count presale")
}
var list []dto.HousePresaleVO
if err = m.Clone().Page(page, size).OrderDesc("id").Scan(&list); err != nil {
return nil, 0, gerror.Wrap(err, "query presale list")
}
return list, total, nil
}

View File

@ -35,8 +35,7 @@ func RegisterTransaction(i ITransaction) { localTransaction = i }
// List 分页查询成交记录,关联小区名,按成交日期倒序。
func (s *transaction) List(ctx context.Context, page, size int, region, keyword string) ([]dto.HouseTransactionVO, int, error) {
m := dao.HouseTransaction.Ctx(ctx).As("t").
LeftJoin("house_community c", "t.community_id=c.id").
Fields("t.*, c.name AS community_name")
LeftJoin("house_community c", "t.community_id=c.id")
if region != "" {
m = m.Where("c.region", region)
}
@ -48,7 +47,7 @@ func (s *transaction) List(ctx context.Context, page, size int, region, keyword
return nil, 0, gerror.Wrap(err, "count transaction")
}
var list []dto.HouseTransactionVO
if err = m.Clone().Page(page, size).OrderDesc("t.deal_date").Scan(&list); err != nil {
if err = m.Clone().Fields("t.*, c.name AS community_name").Page(page, size).OrderDesc("t.deal_date").Scan(&list); err != nil {
return nil, 0, gerror.Wrap(err, "query transaction list")
}
return list, total, nil

View File

@ -0,0 +1,176 @@
package recruitment
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/gogf/gf/v2/frame/g"
"github.com/gogf/gf/v2/os/gtime"
"service.xpcool.com/internal/dao"
"service.xpcool.com/internal/model/do"
"service.xpcool.com/internal/model/dto"
"service.xpcool.com/internal/model/entity"
)
// barkPayload 对应自建 Bark 服务的 /push 接口入参。
type barkPayload struct {
DeviceKey string `json:"device_key"`
Title string `json:"title"`
Body string `json:"body"`
Group string `json:"group"` // 通知分组
Level string `json:"level"` // 优先级
URL string `json:"url"` // 点击跳转
}
// defaultDeviceKey 读取环境变量注入的兜底设备密钥BARK_DEVICE_KEY -> bark.deviceKey
// 当数据库没有任何订阅记录时,推送与测试推送回退到该密钥,做到「填了 env 即可推送」。
func defaultDeviceKey(ctx context.Context) string {
return g.Cfg().MustGet(ctx, "bark.deviceKey", "").String()
}
// defaultSubscription 构造一个使用兜底密钥的默认订阅(全地区/全分类/仅新公告)。
func defaultSubscription(ctx context.Context) *entity.PushSubscription {
key := defaultDeviceKey(ctx)
if key == "" {
return nil
}
return &entity.PushSubscription{
Id: 0, Name: "默认(Bark环境变量)", DeviceKey: key,
Regions: "[]", Categories: "[]", OnlyNew: 1, PushTime: "08:00", Enabled: 1,
}
}
// barkPush 向指定设备密钥发送 Bark 推送自建服务POST {baseUrl}/push
func barkPush(ctx context.Context, deviceKey, title, body string) (bool, string, error) {
baseURL := g.Cfg().MustGet(ctx, "bark.baseUrl", "").String()
if baseURL == "" {
return false, "未配置 bark.baseUrl", fmt.Errorf("bark baseUrl empty")
}
if deviceKey == "" {
// 未显式传入则回退到环境变量兜底密钥。
deviceKey = defaultDeviceKey(ctx)
}
if deviceKey == "" {
return false, "设备密钥为空", fmt.Errorf("device key empty")
}
payload := barkPayload{
DeviceKey: deviceKey,
Title: title,
Body: body,
Group: "recruit_daily",
Level: "active",
}
client := g.Client()
client.SetTimeout(10 * time.Second)
resp, err := client.Post(ctx, strings.TrimRight(baseURL, "/")+"/push", payload)
if err != nil {
return false, err.Error(), err
}
defer resp.Close()
var out struct {
Code int `json:"code"`
Message string `json:"message"`
}
_ = json.Unmarshal(resp.ReadAll(), &out)
ok := resp.StatusCode == 200 && out.Code == 200
msg := out.Message
if msg == "" {
msg = fmt.Sprintf("HTTP %d", resp.StatusCode)
}
return ok, msg, nil
}
// sendDailyDigest 对所有启用订阅发送「招聘早报」汇总(自建 Bark
// 若库内无任何订阅则回退到环境变量兜底密钥BARK_DEVICE_KEY发一份全量早报。
func sendDailyDigest(ctx context.Context) {
var subs []entity.PushSubscription
if err := dao.PushSubscription.Ctx(ctx).Where("enabled", 1).Scan(&subs); err != nil {
g.Log().Errorf(ctx, "load subscriptions failed: %v", err)
return
}
if len(subs) == 0 {
def := defaultSubscription(ctx)
if def == nil {
g.Log().Infof(ctx, "no subscription and no default bark device key, skip digest")
return
}
subs = []entity.PushSubscription{*def}
}
for _, sub := range subs {
title, body := buildDigest(ctx, sub)
if title == "" && body == "" {
// 仅推送新公告且今日无新:跳过,避免打扰。
continue
}
ok, msg, _ := barkPush(ctx, sub.DeviceKey, title, body)
recordPushLog(ctx, sub.Id, title, body, ok, msg)
}
}
// buildDigest 按订阅过滤条件生成早报内容;仅推送新公告且无新时返回空串。
func buildDigest(ctx context.Context, sub entity.PushSubscription) (string, string) {
regions := parseJSONStrings(sub.Regions)
cats := parseJSONInts(sub.Categories)
since := gtime.Now().StartOfDay().String()
m := dao.RecruitmentInfo.Ctx(ctx).Where("status", g.Slice{0, 1}).Where("created_at >= ?", since)
if len(regions) > 0 {
m = m.Where("region IN (?)", regions)
}
if len(cats) > 0 {
m = m.Where("category IN (?)", cats)
}
var list []dto.RecruitmentInfoVO
if err := m.Fields("title, category, region, deadline").OrderDesc("publish_date").Scan(&list); err != nil {
g.Log().Errorf(ctx, "build digest query failed: %v", err)
return "", ""
}
if sub.OnlyNew == 1 && len(list) == 0 {
return "", "" // 无新公告则跳过
}
counts := map[int]int{}
for _, v := range list {
counts[v.Category]++
}
var sb strings.Builder
sb.WriteString("今日新增:")
parts := make([]string, 0, len(counts))
for c, n := range counts {
parts = append(parts, dto.CategoryName(c)+" "+itoa(n))
}
sb.WriteString(strings.Join(parts, " · "))
sb.WriteString("\n")
limit := 8
for i, v := range list {
if i >= limit {
break
}
line := "· " + v.Title
if v.Deadline != "" {
line += "(截止 " + v.Deadline + ""
}
sb.WriteString(line + "\n")
}
title := "贵州招聘早报 · " + gtime.Now().Format("Y-m-d")
if len(regions) == 1 {
title += " · " + regions[0]
}
return title, strings.TrimRight(sb.String(), "\n")
}
// recordPushLog 写推送记录。
func recordPushLog(ctx context.Context, subId uint64, title, body string, ok bool, msg string) {
res := 0
if ok {
res = 1
}
if _, err := dao.PushLog.Ctx(ctx).Data(do.PushLog{
SubscriptionId: subId, PushAt: gtime.Now(), Title: title, Body: body,
Result: res, Error: msg, CreatedAt: gtime.Now(),
}).Insert(); err != nil {
g.Log().Errorf(ctx, "write push log failed: %v", err)
}
}

View File

@ -0,0 +1,596 @@
package recruitment
import (
"context"
"crypto/md5"
"encoding/json"
"fmt"
"net/url"
"regexp"
"strings"
"time"
"github.com/gogf/gf/v2/errors/gerror"
"github.com/gogf/gf/v2/frame/g"
"github.com/gogf/gf/v2/os/gtime"
"service.xpcool.com/internal/dao"
"service.xpcool.com/internal/model/do"
"service.xpcool.com/internal/model/dto"
"service.xpcool.com/internal/model/entity"
)
// crawlerRun 按数据源类型分发抓取,返回运行结果(供写日志/统计)。
func crawlerRun(ctx context.Context, src entity.CrawlSource, force bool) dto.CrawlRunResult {
res := dto.CrawlRunResult{SourceId: src.Id}
defer func() {
// 防止单源解析panic拖垮整体调度。
if r := recover(); r != nil {
res.Err = gerror.Newf("crawler panic: %v", r)
}
}()
var (
infos []dto.RecruitmentInput
err error
)
switch src.SourceType {
case 1: // 静态列表页
infos, err = genericStaticCrawl(ctx, src, force)
case 2: // SPA / JS 渲染(需底层 JSON 接口,首批 POC
infos, err = spaCrawl(ctx, src, force)
case 3: // 需登录/验证码,首批已排除
err = gerror.New("该源需登录/验证码,首批已排除")
case 4: // 附件型,先按静态抓取,附件解析为后续增强
infos, err = genericStaticCrawl(ctx, src, force)
default:
err = gerror.Newf("未知源类型 %d", src.SourceType)
}
res.Err = err
res.Fetched = len(infos)
if err != nil {
return res
}
newCount, updated := 0, 0
for i := range infos {
isNew, e := upsertInfo(ctx, infos[i])
if e != nil {
res.Err = e
continue
}
if isNew {
newCount++
} else {
updated++
}
}
res.NewCount = newCount
res.UpdatedCount = updated
return res
}
// genericStaticCrawl 通用静态列表抓取:取列表页锚点 → 逐条抓详情 → 抽取字段。
func genericStaticCrawl(ctx context.Context, src entity.CrawlSource, force bool) ([]dto.RecruitmentInput, error) {
listURL := joinURL(src.BaseUrl, src.ListPath)
if listURL == "" {
listURL = src.BaseUrl
}
html, err := fetchHTML(ctx, listURL)
if err != nil {
return nil, err
}
g.Log().Infof(ctx, "[recruit-debug] src=%d listURL=%s htmlLen=%d", src.Id, listURL, len(html))
raw := extractAnchors(html, listURL)
anchors := filterArticleAnchors(raw)
g.Log().Infof(ctx, "[recruit-debug] src=%d rawAnchors=%d filtered=%d", src.Id, len(raw), len(anchors))
var out []dto.RecruitmentInput
limit := 60
if force {
limit = 300 // 全量回溯放宽上限
}
for _, a := range anchors {
if len(out) >= limit {
break
}
dHtml, e := fetchHTML(ctx, a.URL)
if e != nil {
continue
}
title, content, pub, dl, ex := extractDetail(dHtml)
if title == "" {
title = a.Text
}
pubDate := normalizeDate(pub)
// 增量模式:跳过 30 天前的公告,避免无效回扫。
if !force && pubDate != "" {
if t, e2 := time.Parse("2006-01-02", pubDate); e2 == nil {
if time.Since(t) > 30*24*time.Hour {
continue
}
}
}
out = append(out, dto.RecruitmentInput{
Title: title,
SourceId: src.Id,
OrgName: src.Name,
Category: src.Category,
Region: src.Region,
PublishDate: pubDate,
Deadline: normalizeDate(dl),
ExamDate: normalizeDate(ex),
Url: a.URL,
Content: content,
Status: 0,
})
}
return out, nil
}
// spaCrawl SPA 源抓取(最佳实践:逆向底层 JSON 接口)。当前为占位实现,
// 若服务端渲染无锚点则返回明确错误便于在阶段0 POC 中补齐接口。
func spaCrawl(ctx context.Context, src entity.CrawlSource, force bool) ([]dto.RecruitmentInput, error) {
listURL := joinURL(src.BaseUrl, src.ListPath)
if listURL == "" {
listURL = src.BaseUrl
}
html, err := fetchHTML(ctx, listURL)
if err != nil {
return nil, err
}
// 部分 SPA 会在 HTML 内联 __NEXT_DATA__ / 初始 state可从中抽取。
if data := extractFromInlineJSON(html); len(data) > 0 {
return data, nil
}
// 纯客户端渲染hash 路由)无服务端内容,需 POC 接入接口。
if len(extractAnchors(html, listURL)) == 0 {
return nil, gerror.New("SPA 无服务端渲染内容,需接入底层 JSON 接口阶段0 POC")
}
return genericStaticCrawl(ctx, src, force)
}
// extractFromInlineJSON 从 SPA 内联 JSON__NEXT_DATA__ / window.__INITIAL_STATE__抽取标题与链接。
// 适配 Vue/Next 等 SSR/CSR 混合页面,作为 SPA 源的兜底解析。
func extractFromInlineJSON(html string) []dto.RecruitmentInput {
var out []dto.RecruitmentInput
// 匹配常见内联 JSON 块,逐块扫描其中的标题/链接字段。
blocks := inlineJSONRe.FindAllStringSubmatch(html, -1)
for _, b := range blocks {
raw := b[1]
var generic map[string]any
if err := json.Unmarshal([]byte(raw), &generic); err != nil {
continue
}
walkJSON(generic, &out)
}
return out
}
// walkJSON 递归遍历内联 JSON提取含标题与链接的条目最佳实践避免硬规则
func walkJSON(node any, out *[]dto.RecruitmentInput) {
switch v := node.(type) {
case map[string]any:
title, _ := v["title"].(string)
link, _ := v["url"].(string)
if link == "" {
link, _ = v["href"].(string)
}
if title != "" && link != "" {
*out = append(*out, dto.RecruitmentInput{
Title: title, Url: link, Status: 0,
})
}
for _, val := range v {
walkJSON(val, out)
}
case []any:
for _, item := range v {
walkJSON(item, out)
}
}
}
// fetchHTML 带超时与浏览器 UA 的请求,降低被反爬概率。
func fetchHTML(ctx context.Context, u string) (string, error) {
client := g.Client()
client.SetTimeout(15 * time.Second)
client.SetHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36")
client.SetHeader("Accept", "text/html,application/xhtml+xml,*/*")
resp, err := client.Get(ctx, u)
if err != nil {
return "", gerror.Wrapf(err, "fetch %s", u)
}
defer resp.Close()
if resp.StatusCode != 200 {
return "", gerror.Newf("HTTP %d for %s", resp.StatusCode, u)
}
return resp.ReadAllString(), nil
}
type anchor struct {
URL string
Text string
}
var (
// anchorBlockRe 匹配完整 <a>...</a> 块hrefRe/titleAttrRe 从块内分别抽取链接与 title 属性。
anchorBlockRe = regexp.MustCompile(`(?is)<a\b[^>]*>([\s\S]*?)</a>`)
hrefRe = regexp.MustCompile(`(?i)\bhref\s*=\s*["']([^"']+)["']`)
titleAttrRe = regexp.MustCompile(`(?i)\btitle\s*=\s*["']([^"']+)["']`)
tagRe = regexp.MustCompile(`<[^>]+>`)
scriptRe = regexp.MustCompile(`(?is)<script[\s\S]*?</script>`)
styleRe = regexp.MustCompile(`(?is)<style[\s\S]*?</style>`)
titleRe = regexp.MustCompile(`(?is)<title[^>]*>([\s\S]*?)</title>`)
h1Re = regexp.MustCompile(`(?is)<h1[^>]*>([\s\S]*?)</h1>`)
dateRe = regexp.MustCompile(`(\d{4})[-/年.](\d{1,2})[-/月.](\d{1,2})`)
inlineJSONRe = regexp.MustCompile(`(?is)(?:__NEXT_DATA__|__INITIAL_STATE__|window\.__[A-Z_]+)\s*=\s*(\{[\s\S]*?\})\s*;?`)
spaceRe = regexp.MustCompile(`\s+`)
// recruitKw 招聘类公告常见语义词,用于从政府站链接中识别公告条目。
recruitKw = regexp.MustCompile(`招聘|招考|公招|选聘|引进|人才|公告|公示|录用|录取|面试|笔试|报名|选调|遴选|拟聘|聘用|招募|招录|考试|考录|体检|考察|资格复审|递补`)
)
// extractAnchors 从 HTML 抽取绝对化后的锚点URL + 文本)。
// 政府站常把真实标题放在 title 属性、inner text 仅"详细/更多",故 inner 过短时回退 title。
func extractAnchors(html, base string) []anchor {
out := make([]anchor, 0)
for _, block := range anchorBlockRe.FindAllStringSubmatch(html, -1) {
full := block[0]
hm := hrefRe.FindStringSubmatch(full)
if len(hm) < 2 {
continue
}
href := strings.TrimSpace(hm[1])
if href == "" || strings.HasPrefix(href, "javascript:") ||
strings.HasPrefix(href, "#") || strings.HasPrefix(href, "mailto:") {
continue
}
inner := strings.TrimSpace(stripTags(block[1]))
text := inner
if tm := titleAttrRe.FindStringSubmatch(full); len(tm) >= 2 {
t := strings.TrimSpace(tm[1])
if len([]rune(text)) < 4 && t != "" {
text = t
}
}
if text == "" {
continue
}
abs := resolveURL(base, href)
if abs == "" {
continue
}
out = append(out, anchor{URL: abs, Text: text})
}
return out
}
// filterArticleAnchors 过滤导航/无用链接,保留疑似招聘公告条目并去重。
func filterArticleAnchors(as []anchor) []anchor {
out := make([]anchor, 0, len(as))
for _, a := range as {
text := strings.TrimSpace(a.Text)
if len([]rune(text)) < 4 {
continue
}
u := strings.ToLower(a.URL)
// URL 形态线索:静态详情页、政务公开/人事招考栏目、含年份。
urlHint := strings.Contains(u, ".html") || strings.Contains(u, ".shtml") ||
strings.Contains(u, ".php") || strings.Contains(u, "/tzgg") ||
strings.Contains(u, "/rszk") || strings.Contains(u, "/rsxx") ||
strings.Contains(u, "/zfxx") || dateRe.MatchString(a.URL)
// 文本语义线索:政府站公告标题常含这些词(必须命中,过滤"市长信箱/重点领域"等非招聘页)。
textHint := recruitKw.MatchString(text) || recruitKw.MatchString(a.URL)
// 同时满足"内容页形态"与"招聘语义",避免误抓栏目/互动页。
if urlHint && textHint {
out = append(out, a)
}
}
seen := map[string]bool{}
uniq := out[:0]
for _, a := range out {
if seen[a.URL] {
continue
}
seen[a.URL] = true
uniq = append(uniq, a)
}
return uniq
}
// extractDetail 从详情页抽取标题/正文/发布日期/报名截止/笔试日期(启发式,可按源调优)。
func extractDetail(html string) (title, content, publishDate, deadline, examDate string) {
if m := titleRe.FindStringSubmatch(html); len(m) > 1 {
title = stripTags(m[1])
}
if title == "" {
if m := h1Re.FindStringSubmatch(html); len(m) > 1 {
title = stripTags(m[1])
}
}
title = strings.TrimSpace(title)
body := scriptRe.ReplaceAllString(html, " ")
body = styleRe.ReplaceAllString(body, " ")
text := stripTags(body)
text = collapseSpace(text)
if len([]rune(text)) > 4000 {
text = string([]rune(text)[:4000])
}
content = text
publishDate = normalizeDate(firstDate(html))
deadline = normalizeDate(findDateAfter(text, "报名"))
examDate = normalizeDate(findDateAfter(text, "笔试"))
return
}
// firstDate 返回文本中首个日期(归一化后)。
func firstDate(s string) string {
m := dateRe.FindStringSubmatch(s)
if len(m) == 0 {
return ""
}
return normalizeDate(m[0])
}
// findDateAfter 在 keyword 之后查找下一个日期(用于报名/笔试时间)。
func findDateAfter(text, keyword string) string {
idx := strings.Index(text, keyword)
if idx < 0 {
return ""
}
rest := text[idx:]
m := dateRe.FindStringSubmatch(rest)
if len(m) == 0 {
return ""
}
return normalizeDate(m[0])
}
// normalizeDate 将多种中文/西式日期归一化为 YYYY-MM-DD。
func normalizeDate(s string) string {
s = strings.TrimSpace(s)
if s == "" {
return ""
}
m := dateRe.FindStringSubmatch(s)
if len(m) == 0 {
return ""
}
y, mo, d := m[1], atoiSafe(m[2]), atoiSafe(m[3])
return fmt.Sprintf("%s-%02d-%02d", y, mo, d)
}
// stripTags 去除 HTML 标签。
func stripTags(s string) string {
return tagRe.ReplaceAllString(s, " ")
}
// collapseSpace 折叠空白字符。
func collapseSpace(s string) string {
return spaceRe.ReplaceAllString(s, " ")
}
// resolveURL 将相对链接解析为绝对 URL。
func resolveURL(base, href string) string {
if strings.HasPrefix(href, "http://") || strings.HasPrefix(href, "https://") {
return href
}
u, err := url.Parse(base)
if err != nil {
return ""
}
ref, err := url.Parse(href)
if err != nil {
return ""
}
return u.ResolveReference(ref).String()
}
// joinURL 拼接基础域名与路径。
func joinURL(base, p string) string {
if p == "" {
return base
}
if strings.HasPrefix(p, "http") {
return p
}
u, err := url.Parse(base)
if err != nil {
return base + p
}
ref, err := url.Parse(p)
if err != nil {
return base + p
}
return u.ResolveReference(ref).String()
}
func atoiSafe(s string) int {
n := 0
for _, c := range s {
if c < '0' || c > '9' {
break
}
n = n*10 + int(c-'0')
}
return n
}
// ---------- 去重与写入 ----------
func fingerprintOf(in dto.RecruitmentInput) string {
h := md5.Sum([]byte(fmt.Sprintf("%d|%s|%s|%s", in.SourceId, in.Title, in.PublishDate, in.Url)))
return fmt.Sprintf("%x", h)
}
func groupKeyOf(in dto.RecruitmentInput) string {
h := md5.Sum([]byte(fmt.Sprintf("%s|%s", in.Title, in.PublishDate)))
return fmt.Sprintf("%x", h)
}
// upsertInfo 按指纹去重写入公告;已存在则更新可变字段(内容/状态/日期)。
func upsertInfo(ctx context.Context, in dto.RecruitmentInput) (bool, error) {
if in.Fingerprint == "" {
in.Fingerprint = fingerprintOf(in)
}
if in.GroupKey == "" {
in.GroupKey = groupKeyOf(in)
}
var exist entity.RecruitmentInfo
if err := dao.RecruitmentInfo.Ctx(ctx).Where("fingerprint", in.Fingerprint).Scan(&exist); err != nil {
// gf 的 Scan 在查无记录时返回 "no rows" 错误,视为未存在,继续插入。
if !strings.Contains(err.Error(), "no rows") {
return false, gerror.Wrap(err, "query exist info")
}
}
if exist.Id > 0 {
_, err := dao.RecruitmentInfo.Ctx(ctx).Where("id", exist.Id).Data(do.RecruitmentInfo{
Content: in.Content, Status: in.Status, Deadline: in.Deadline,
ExamDate: in.ExamDate, UpdatedAt: gtime.Now(),
}).Update()
if err != nil {
return false, gerror.Wrap(err, "update info")
}
return false, nil
}
orgId, err := upsertOrg(ctx, in.OrgName, in.Category, in.Region)
if err != nil {
return false, err
}
in.OrgId = orgId
if _, err = dao.RecruitmentInfo.Ctx(ctx).Data(toRecruitmentDO(in)).InsertAndGetId(); err != nil {
return false, gerror.Wrap(err, "insert info")
}
return true, nil
}
// upsertOrg 发布主体按名称去重,返回主键。
func upsertOrg(ctx context.Context, name string, category int, region string) (uint64, error) {
if name == "" {
return 0, nil
}
var org entity.Organization
if err := dao.Organization.Ctx(ctx).Where("name", name).Scan(&org); err != nil {
if !strings.Contains(err.Error(), "no rows") {
return 0, gerror.Wrap(err, "query org")
}
}
if org.Id > 0 {
return org.Id, nil
}
id, err := dao.Organization.Ctx(ctx).Data(do.Organization{
Name: name, Type: mapCategoryToOrgType(category), Region: region,
CreatedAt: gtime.Now(), UpdatedAt: gtime.Now(),
}).InsertAndGetId()
if err != nil {
return 0, gerror.Wrap(err, "insert org")
}
return uint64(id), nil
}
func mapCategoryToOrgType(c int) int {
switch c {
case 2:
return 2 // 事业单位
case 3:
return 3 // 国企
case 4:
return 4 // 央企
case 5:
return 5 // 私企
default:
return 6
}
}
func toRecruitmentDO(in dto.RecruitmentInput) do.RecruitmentInfo {
now := gtime.Now()
return do.RecruitmentInfo{
Title: in.Title, SourceId: in.SourceId, OrgId: in.OrgId, OrgName: in.OrgName,
Category: in.Category, Region: in.Region, PublishDate: nullIfEmpty(in.PublishDate),
Deadline: nullIfEmpty(in.Deadline), ExamDate: nullIfEmpty(in.ExamDate), Url: in.Url, Content: in.Content,
Attachments: in.Attachments, Status: in.Status, Fingerprint: in.Fingerprint,
GroupKey: in.GroupKey, CreatedAt: now, UpdatedAt: now,
}
}
// nullIfEmpty 将空字符串转为 nil便于插入 NULLDATE 等可空列不接受空串)。
func nullIfEmpty(s string) any {
if strings.TrimSpace(s) == "" {
return nil
}
return s
}
// recordCrawlLog 写抓取日志并更新数据源运行状态(成功清失败计数;失败累加)。
func recordCrawlLog(ctx context.Context, res dto.CrawlRunResult) {
errMsg := ""
if res.Err != nil {
// 错误可能含整页 SQL截断避免超过 error 列长度导致日志也写不进。
errMsg = res.Err.Error()
const maxErr = 900
if len(errMsg) > maxErr {
errMsg = errMsg[:maxErr] + "...(truncated)"
}
}
if _, err := dao.CrawlLog.Ctx(ctx).Data(do.CrawlLog{
SourceId: res.SourceId, RunAt: gtime.Now(), Fetched: res.Fetched,
NewCount: res.NewCount, UpdatedCount: res.UpdatedCount, Error: errMsg,
}).Insert(); err != nil {
g.Log().Errorf(ctx, "write crawl log failed: %v", err)
}
if res.Err != nil {
var src entity.CrawlSource
_ = dao.CrawlSource.Ctx(ctx).Where("id", res.SourceId).Scan(&src)
_, _ = dao.CrawlSource.Ctx(ctx).Where("id", res.SourceId).
Data(do.CrawlSource{FailCount: src.FailCount + 1}).Update()
return
}
_, _ = dao.CrawlSource.Ctx(ctx).Where("id", res.SourceId).
Data(do.CrawlSource{LastSuccessAt: gtime.Now().String(), FailCount: 0}).Update()
}
// ---------- 订阅辅助 ----------
// loadSubscription 加载推送订阅id>0 按 id否则取首个启用订阅。
func loadSubscription(ctx context.Context, id uint64) (*entity.PushSubscription, error) {
var sub entity.PushSubscription
m := dao.PushSubscription.Ctx(ctx)
if id > 0 {
m = m.Where("id", id)
} else {
m = m.Where("enabled", 1)
}
if err := m.OrderDesc("id").Scan(&sub); err != nil {
return nil, gerror.Wrap(err, "query subscription")
}
if sub.Id == 0 {
return nil, nil
}
return &sub, nil
}
func doPushSubscription(in dto.SubscriptionInput) do.PushSubscription {
regions, _ := json.Marshal(in.Regions)
cats, _ := json.Marshal(in.Categories)
return do.PushSubscription{
Name: in.Name, DeviceKey: in.DeviceKey, Regions: string(regions),
Categories: string(cats), OnlyNew: in.OnlyNew, PushTime: in.PushTime,
Enabled: in.Enabled, UpdatedAt: gtime.Now(),
}
}
func parseJSONStrings(s string) []string {
if s == "" {
return nil
}
var out []string
_ = json.Unmarshal([]byte(s), &out)
return out
}
func parseJSONInts(s string) []int {
if s == "" {
return nil
}
var out []int
_ = json.Unmarshal([]byte(s), &out)
return out
}

View File

@ -263,7 +263,11 @@ func (s *recruitment) PushTest(ctx context.Context, subId uint64, title, body st
return false, err.Error(), err
}
if sub == nil {
return false, "无启用的推送订阅", gerror.New("no enabled subscription")
// 未指定/未找到订阅时回退到环境变量兜底密钥BARK_DEVICE_KEY
sub = defaultSubscription(ctx)
}
if sub == nil {
return false, "无启用的推送订阅且未配置默认设备密钥", gerror.New("no enabled subscription")
}
if title == "" {
title = "招聘聚合 · 推送测试"

View File

@ -0,0 +1,28 @@
package recruitment
import (
"context"
"github.com/gogf/gf/v2/frame/g"
"github.com/gogf/gf/v2/os/gcron"
)
// StartScheduler 注册定时任务:每日 03:00 增量抓取全部启用源08:00 推送招聘早报。
// 调度器随 service 进程常驻,不额外引入系统 crontab。
func StartScheduler(ctx context.Context) {
if _, err := gcron.Add(ctx, "0 3 * * *", func(c context.Context) {
_, summary, e := Recruitment().Trigger(c, 0, false)
if e != nil {
g.Log().Errorf(c, "recruit daily crawl failed: %v", e)
} else {
g.Log().Infof(c, "recruit daily crawl done: %s", summary)
}
}, "recruit-crawl-daily"); err != nil {
g.Log().Errorf(ctx, "add recruit crawl cron failed: %v", err)
}
if _, err := gcron.Add(ctx, "0 8 * * *", func(c context.Context) {
sendDailyDigest(c)
}, "recruit-push-daily"); err != nil {
g.Log().Errorf(ctx, "add recruit push cron failed: %v", err)
}
}

View File

@ -33,6 +33,16 @@ var AdminOperationLog = map[string]*gdb.TableField{
Extra: "",
Comment: "",
},
"admin_username": {
Index: 2,
Name: "admin_username",
Type: "varchar(64)",
Null: false,
Key: "MUL",
Default: "",
Extra: "",
Comment: "管理员账号(冗余)",
},
"permission": {
Index: 2,
Name: "permission",
@ -73,6 +83,16 @@ var AdminOperationLog = map[string]*gdb.TableField{
Extra: "",
Comment: "",
},
"ip_location": {
Index: 6,
Name: "ip_location",
Type: "varchar(255)",
Null: false,
Key: "MUL",
Default: "",
Extra: "",
Comment: "IP 归属地(省/市/运营商)",
},
"request_param": {
Index: 6,
Name: "request_param",
@ -103,6 +123,26 @@ var AdminOperationLog = map[string]*gdb.TableField{
Extra: "",
Comment: "",
},
"error_message": {
Index: 9,
Name: "error_message",
Type: "varchar(512)",
Null: true,
Key: "",
Default: nil,
Extra: "",
Comment: "失败原因摘要",
},
"user_agent": {
Index: 10,
Name: "user_agent",
Type: "varchar(512)",
Null: false,
Key: "",
Default: "",
Extra: "",
Comment: "浏览器 UA",
},
"created_at": {
Index: 9,
Name: "created_at",

View File

@ -10,8 +10,14 @@ logger:
database:
default:
link: "${DB_DSN}"
recruitment:
link: "${RECRUITMENT_DB_DSN}"
jwt:
# 生产环境必须通过 JWT_SECRET 覆盖该值
secret: "${JWT_SECRET}"
accessExpire: "2h"
refreshExpire: "720h"
bark:
baseUrl: "${BARK_BASE_URL}"
deviceKey: "${BARK_DEVICE_KEY}"
pushTime: "${BARK_PUSH_TIME}"

View File

@ -1,4 +1,5 @@
server: { address: ":10100", openapiPath: "/api.json", swaggerPath: "/swagger" }
logger: { level: "warning", stdout: true }
database: { default: { link: "${DB_DSN}" } }
database: { default: { link: "${DB_DSN}" }, recruitment: { link: "${RECRUITMENT_DB_DSN}" } }
jwt: { secret: "${JWT_SECRET}", accessExpire: "2h", refreshExpire: "720h" }
bark: { baseUrl: "${BARK_BASE_URL}", deviceKey: "${BARK_DEVICE_KEY}", pushTime: "${BARK_PUSH_TIME}" }

View File

@ -1,3 +1,4 @@
server: { address: ":10100" }
database: { default: { link: "${TEST_DB_DSN}" } }
database: { default: { link: "${TEST_DB_DSN}" }, recruitment: { link: "${RECRUITMENT_DB_DSN}" } }
jwt: { secret: "${JWT_SECRET}", accessExpire: "15m", refreshExpire: "1h" }
bark: { baseUrl: "${BARK_BASE_URL}", deviceKey: "${BARK_DEVICE_KEY}", pushTime: "${BARK_PUSH_TIME}" }

View File

@ -24,6 +24,26 @@ CREATE TABLE IF NOT EXISTS admin_role_menu (
PRIMARY KEY (id), UNIQUE KEY uk_role_menu (role_id,menu_id), KEY idx_arm_menu (menu_id), KEY idx_arm_deleted_at (deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Role-menu relation';
CREATE TABLE IF NOT EXISTS admin_operation_log (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, admin_user_id BIGINT UNSIGNED NOT NULL, permission VARCHAR(128) NOT NULL DEFAULT '', method VARCHAR(12) NOT NULL, path VARCHAR(255) NOT NULL, ip VARCHAR(64) NOT NULL DEFAULT '', request_param JSON NULL, duration_ms INT UNSIGNED NOT NULL DEFAULT 0, status_code INT NOT NULL DEFAULT 0, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, deleted_at DATETIME NULL,
PRIMARY KEY (id), KEY idx_aol_user_created (admin_user_id,created_at), KEY idx_aol_permission_created (permission,created_at), KEY idx_aol_deleted_at (deleted_at)
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
admin_user_id BIGINT UNSIGNED NOT NULL COMMENT '管理员ID',
admin_username VARCHAR(64) NOT NULL DEFAULT '' COMMENT '管理员账号冗余便于检索与展示',
permission VARCHAR(128) NOT NULL DEFAULT '' COMMENT '权限码',
method VARCHAR(12) NOT NULL COMMENT 'HTTP 方法',
path VARCHAR(255) NOT NULL COMMENT '请求路径',
ip VARCHAR(64) NOT NULL DEFAULT '' COMMENT '来源 IP',
ip_location VARCHAR(255) NOT NULL DEFAULT '' COMMENT 'IP 归属地//运营商离线解析',
request_param JSON NULL COMMENT '请求参数敏感字段已脱敏',
duration_ms INT UNSIGNED NOT NULL DEFAULT 0 COMMENT '耗时(ms)',
status_code INT NOT NULL DEFAULT 0 COMMENT 'HTTP 状态码',
error_message VARCHAR(512) NULL COMMENT '失败原因摘要',
user_agent VARCHAR(512) NOT NULL DEFAULT '' COMMENT '浏览器 UA',
created_at DATETIME NOT NULL,
updated_at DATETIME NOT NULL,
deleted_at DATETIME NULL,
PRIMARY KEY (id),
KEY idx_aol_user_created (admin_user_id,created_at),
KEY idx_aol_permission_created (permission,created_at),
KEY idx_aol_ip (ip),
KEY idx_aol_username (admin_username),
KEY idx_aol_deleted_at (deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Admin operation audit log';

View File

@ -17,6 +17,7 @@ CREATE TABLE IF NOT EXISTS house_community (
households INT NOT NULL DEFAULT 0 COMMENT '总户数',
plot_ratio DECIMAL(6,3) NOT NULL DEFAULT 0 COMMENT '容积率',
green_rate DECIMAL(6,3) NOT NULL DEFAULT 0 COMMENT '绿化率',
avg_price DECIMAL(10,2) NOT NULL DEFAULT 0 COMMENT '小区参考均价(/平米)',
property_company VARCHAR(128) NOT NULL DEFAULT '' COMMENT '物业公司',
property_fee DECIMAL(10,2) NOT NULL DEFAULT 0 COMMENT '物业费(//平米)',
developer VARCHAR(128) NOT NULL DEFAULT '' COMMENT '开发商',

View File

@ -0,0 +1,12 @@
-- 010 服务器日志操作审计增强补充 IP 归属地管理员账号失败原因UA 等检索维度
-- 仅对已按 001_core 建表但尚未包含新列的存量库执行全新环境直接由 001_core 建表无需本文件
ALTER TABLE admin_operation_log
ADD COLUMN admin_username VARCHAR(64) NOT NULL DEFAULT '' COMMENT '管理员账号冗余便于检索与展示' AFTER admin_user_id,
ADD COLUMN ip_location VARCHAR(255) NOT NULL DEFAULT '' COMMENT 'IP 归属地//运营商离线解析' AFTER ip,
ADD COLUMN error_message VARCHAR(512) NULL COMMENT '失败原因摘要' AFTER status_code,
ADD COLUMN user_agent VARCHAR(512) NOT NULL DEFAULT '' COMMENT '浏览器 UA' AFTER error_message;
ALTER TABLE admin_operation_log
ADD KEY idx_aol_ip (ip),
ADD KEY idx_aol_username (admin_username);

View File

@ -0,0 +1,24 @@
-- 013_house_presale.sql
-- 新房预售许可证公示数据源贵阳市住建局商品房预售许可证公示政府公开数据
CREATE TABLE IF NOT EXISTS house_presale (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
presale_no VARCHAR(64) NOT NULL DEFAULT '' COMMENT '预售证号',
community_name VARCHAR(128) NOT NULL COMMENT '楼盘/项目名',
developer VARCHAR(128) NOT NULL DEFAULT '' COMMENT '房地产开发企业',
region VARCHAR(64) NOT NULL DEFAULT '' COMMENT '区县',
address VARCHAR(255) NOT NULL DEFAULT '' COMMENT '建筑位置',
building_no VARCHAR(64) NOT NULL DEFAULT '' COMMENT '楼栋号',
house_count INT NOT NULL DEFAULT 0 COMMENT '套数',
area DECIMAL(12,2) NOT NULL DEFAULT 0 COMMENT '建筑面积(平米)',
purpose VARCHAR(32) NOT NULL DEFAULT '' COMMENT '规划用途(住宅/商业)',
issue_date VARCHAR(32) NOT NULL DEFAULT '' COMMENT '初始核发日期',
source VARCHAR(32) NOT NULL DEFAULT 'gov_presale' COMMENT '数据来源',
created_at DATETIME NOT NULL,
updated_at DATETIME NOT NULL,
deleted_at DATETIME NULL,
PRIMARY KEY (id),
UNIQUE KEY uk_presale_no (presale_no),
KEY idx_presale_region (region),
KEY idx_presale_deleted_at (deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='新房预售许可证公示';

View File

@ -0,0 +1,17 @@
-- 013_workbench_menu.sql
-- 2026-08-27 新增工作台顶层菜单RBAC 动态路由
-- 页面views/dashboard/workbench/index.vue复用现有列表接口人员/角色/菜单树/操作日志/登录日志
-- 在前端聚合展示无需新增后端接口数据权限沿用各模块既有按钮权限超管全量可见
-- 幂等INSERT ... ON DUPLICATE KEY UPDATE + INSERT IGNORE可重复执行
-- 依赖004_seed.sqladmin_menu / admin_role_menu 结构 + 超管角色 role_id=1
-- ============ 1) 工作台菜单type=1顶层sort=0 置顶 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(9, 0, '工作台', 'mdi:view-dashboard', 1, '/workbench', 'dashboard/workbench/index', 'workbench', 0, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 2) 超管角色role_id=1绑定工作台菜单 ============
INSERT IGNORE INTO admin_role_menu (role_id, menu_id, created_at, updated_at)
SELECT 1, id, NOW(), NOW() FROM admin_menu WHERE id = 9;

View File

@ -0,0 +1,20 @@
-- 014_house_presale_menu.sql
-- 新房预售独立菜单 + 权限种子
-- 幂等INSERT ... ON DUPLICATE KEY UPDATE + INSERT IGNORE可重复执行
-- ============ 1) 新房预售菜单type=1挂到看房中心 90 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(95, 90, '新房预售', 'mdi:office-building', 1, 'presale', 'house/presale/index', 'house:presale', 5, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 2) 预售证查询按钮type=2 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(950, 95, '查询', '', 2, 'POST /api/service/admin/house/presale/list', '', 'house:presale:list', 1, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), type=VALUES(type),
path=VALUES(path), permission=VALUES(permission), sort=VALUES(sort), status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 3) 超管角色绑定 ============
INSERT IGNORE INTO admin_role_menu (role_id, menu_id, created_at, updated_at)
SELECT 1, id, NOW(), NOW() FROM admin_menu WHERE id IN (95, 950);

View File

@ -0,0 +1,103 @@
-- 招聘考试聚合模块 · 独立数据库 recruitment 初始化
-- 该库与 service 主库隔离通过 config database.recruitment.link 指定
-- 执行顺序先建库CREATE DATABASE 由部署脚本完成再导入本文件
CREATE TABLE IF NOT EXISTS `organization` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(200) NOT NULL DEFAULT '' COMMENT '主体名称',
`type` TINYINT NOT NULL DEFAULT 6 COMMENT '1机关 2事业 3国企 4央企 5私企 6其他',
`official_site` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '官网',
`region` VARCHAR(50) NOT NULL DEFAULT '' COMMENT '地区',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_org_name` (`name`(100)),
KEY `idx_org_region` (`region`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='发布主体';
CREATE TABLE IF NOT EXISTS `crawl_source` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(200) NOT NULL DEFAULT '' COMMENT '数据源名称',
`base_url` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '站点基础URL',
`source_type` TINYINT NOT NULL DEFAULT 1 COMMENT '1静态 2SPA 3需登录 4附件型',
`category` TINYINT NOT NULL DEFAULT 6 COMMENT '默认考试分类',
`region` VARCHAR(50) NOT NULL DEFAULT '' COMMENT '默认地区',
`list_path` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '列表页路径/接口',
`config` VARCHAR(1000) NOT NULL DEFAULT '' COMMENT '适配器扩展配置(JSON)',
`enabled` TINYINT NOT NULL DEFAULT 1 COMMENT '是否启用 0 1',
`cron_expr` VARCHAR(100) NOT NULL DEFAULT '' COMMENT '调度表达式',
`last_success_at` DATETIME NULL DEFAULT NULL COMMENT '最近成功抓取时间',
`fail_count` INT NOT NULL DEFAULT 0 COMMENT '连续失败次数',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_src_enabled` (`enabled`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='数据源配置';
CREATE TABLE IF NOT EXISTS `recruitment_info` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`title` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '公告标题',
`source_id` BIGINT UNSIGNED NOT NULL DEFAULT 0 COMMENT '数据源ID',
`org_id` BIGINT UNSIGNED NOT NULL DEFAULT 0 COMMENT '发布主体ID',
`org_name` VARCHAR(200) NOT NULL DEFAULT '' COMMENT '发布主体名称(冗余)',
`category` TINYINT NOT NULL DEFAULT 6 COMMENT '1公务员 2事业单位 3国企 4央企 5私企 6其他',
`region` VARCHAR(50) NOT NULL DEFAULT '' COMMENT '地区',
`publish_date` DATE NULL DEFAULT NULL COMMENT '发布日期',
`deadline` DATE NULL DEFAULT NULL COMMENT '报名/截止日期',
`exam_date` DATE NULL DEFAULT NULL COMMENT '笔试日期',
`url` VARCHAR(1000) NOT NULL DEFAULT '' COMMENT '原文链接',
`content` LONGTEXT COMMENT '正文/摘要',
`attachments` VARCHAR(2000) NOT NULL DEFAULT '' COMMENT '附件链接与解析结果(JSON)',
`status` TINYINT NOT NULL DEFAULT 0 COMMENT '0有效 1已更正 2已失效 3已删除',
`fingerprint` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '去重指纹',
`group_key` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '跨源同公告聚合键',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_fingerprint` (`fingerprint`),
KEY `idx_url` (`url`(255)),
KEY `idx_region_category` (`region`,`category`),
KEY `idx_publish_date` (`publish_date`),
KEY `idx_status` (`status`),
KEY `idx_group_key` (`group_key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='招聘考试公告主表';
CREATE TABLE IF NOT EXISTS `crawl_log` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`source_id` BIGINT UNSIGNED NOT NULL DEFAULT 0 COMMENT '数据源ID',
`run_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '运行时间',
`fetched` INT NOT NULL DEFAULT 0 COMMENT '抓取条数',
`new_count` INT NOT NULL DEFAULT 0 COMMENT '新增条数',
`updated_count` INT NOT NULL DEFAULT 0 COMMENT '更新条数',
`error` VARCHAR(1000) NOT NULL DEFAULT '' COMMENT '错误信息',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_log_source` (`source_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='抓取任务日志';
CREATE TABLE IF NOT EXISTS `push_subscription` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(100) NOT NULL DEFAULT '' COMMENT '订阅名称',
`device_key` VARCHAR(200) NOT NULL DEFAULT '' COMMENT 'Bark 设备密钥',
`regions` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '订阅地区(JSON数组)',
`categories` VARCHAR(200) NOT NULL DEFAULT '' COMMENT '订阅分类(JSON数组)',
`only_new` TINYINT NOT NULL DEFAULT 1 COMMENT '仅推送新公告 0 1',
`push_time` VARCHAR(10) NOT NULL DEFAULT '08:00' COMMENT '推送时间 HH:mm',
`enabled` TINYINT NOT NULL DEFAULT 1 COMMENT '是否启用 0 1',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Bark 推送订阅';
CREATE TABLE IF NOT EXISTS `push_log` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`subscription_id` BIGINT UNSIGNED NOT NULL DEFAULT 0 COMMENT '订阅ID',
`push_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '推送时间',
`title` VARCHAR(500) NOT NULL DEFAULT '' COMMENT '推送标题',
`body` TEXT COMMENT '推送内容',
`result` TINYINT NOT NULL DEFAULT 0 COMMENT '1成功 0失败',
`error` VARCHAR(1000) NOT NULL DEFAULT '' COMMENT '错误信息',
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_push_sub` (`subscription_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='推送记录';

View File

@ -0,0 +1,25 @@
-- 招聘考试聚合模块 · 数据源种子
-- 首批范围官方/半官方公务员/事业/国企/央企贵阳两静态源启用其余待阶段0 POC 后启用
-- source_type: 1静态 2SPA 3需登录 4附件型
-- 贵阳市人力资源和社会保障局 · 人事招考静态已核验可抓
INSERT INTO `crawl_source` (`name`, `base_url`, `source_type`, `category`, `region`, `list_path`, `enabled`)
VALUES ('贵阳市人社局-人事招考', 'http://rsj.guiyang.gov.cn', 1, 2, '贵阳',
'/zfxxgk/fdzdgklm/zfxxgkrsxx/rszk/', 1);
-- 贵阳市人民政府 · 人事招考/招聘信息静态已核验可抓
INSERT INTO `crawl_source` (`name`, `base_url`, `source_type`, `category`, `region`, `list_path`, `enabled`)
VALUES ('贵阳市政府-人事招考', 'https://www.guiyang.gov.cn', 1, 2, '贵阳',
'/zwgk/zdlyxxgk2024/shsyjzdms/jycy/rszk/', 1);
-- 贵州人事考试信息网SPA/hash 路由 POC 接入底层 JSON 接口先禁用
INSERT INTO `crawl_source` (`name`, `base_url`, `source_type`, `category`, `region`, `list_path`, `enabled`)
VALUES ('贵州人事考试信息网', 'https://www.gzrsks.com.cn', 2, 1, '省直', '/', 0);
-- 贵州省国资委国资央企招聘平台 POC 确认接口形态先禁用
INSERT INTO `crawl_source` (`name`, `base_url`, `source_type`, `category`, `region`, `list_path`, `enabled`)
VALUES ('贵州国资央企招聘平台', 'https://cujiuye.iguopin.com', 2, 4, '省直', '/', 0);
-- 中国贵州茅台集团官网附件型 POC 接入附件解析先禁用
INSERT INTO `crawl_source` (`name`, `base_url`, `source_type`, `category`, `region`, `list_path`, `enabled`)
VALUES ('贵州茅台集团', 'https://www.moutaichina.com', 4, 4, '省直', '/', 0);

View File

@ -0,0 +1,59 @@
-- 003_menu.sql
-- 招聘考试聚合模块菜单 + API 权限种子
-- 约定 011_house_menu.sql 一致
-- type=1 菜单component 指向 views/<path>/index不含 views/ 前缀与 .vue 后缀
-- permission 用于前端侧边栏可见性
-- type=2 按钮path 为完整 "METHOD /api/service/admin/..."permission RBAC 权限码
-- RBAC 中间件按 method+path 反查该权限码并校验管理员是否持有
-- 依赖先执行 001_init.sql + 002_seed_sources.sqladmin_menu/admin_role_menu 已存在
-- 幂等INSERT ... ON DUPLICATE KEY UPDATE + INSERT IGNORE可重复执行
-- ============ 1) 招聘中心菜单type=1 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(95, 0, '招聘中心', 'mdi:briefcase-search', 1, '/recruitment', '', 'recruitment:center', 7, 1, 0, NOW(), NOW()),
(951, 95, '招聘公告', 'mdi:file-document', 1, 'recruitment/info', 'recruitment/info/index', 'recruitment:info', 1, 1, 0, NOW(), NOW()),
(952, 95, '数据看板', 'mdi:chart-box', 1, 'recruitment/dashboard','recruitment/dashboard/index','recruitment:dashboard', 2, 1, 0, NOW(), NOW()),
(953, 95, '数据源', 'mdi:web-sync', 1, 'recruitment/crawler', 'recruitment/crawler/index', 'recruitment:crawler', 3, 1, 0, NOW(), NOW()),
(954, 95, '推送订阅', 'mdi:bell-ring', 1, 'recruitment/push', 'recruitment/push/index', 'recruitment:push', 4, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 2) 招聘公告按钮type=2 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(9510, 951, '查询', '', 2, 'POST /api/service/admin/recruitment/info/list', '', 'recruitment:info:list', 1, 1, 0, NOW(), NOW()),
(9511, 951, '详情', '', 2, 'POST /api/service/admin/recruitment/info/detail', '', 'recruitment:info:detail', 2, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 3) 数据看板按钮type=2 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(9520, 952, '统计', '', 2, 'POST /api/service/admin/recruitment/stats', '', 'recruitment:dashboard:stats', 1, 1, 0, NOW(), NOW()),
(9521, 952, '趋势', '', 2, 'POST /api/service/admin/recruitment/trend', '', 'recruitment:dashboard:trend', 2, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 4) 数据源按钮type=2 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(9530, 953, '列表', '', 2, 'POST /api/service/admin/recruitment/source/list', '', 'recruitment:crawler:list', 1, 1, 0, NOW(), NOW()),
(9531, 953, '触发', '', 2, 'POST /api/service/admin/recruitment/crawl/trigger', '', 'recruitment:crawler:trigger', 2, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 5) 推送订阅按钮type=2 ============
INSERT INTO admin_menu (id, parent_id, name, icon, type, path, component, permission, sort, status, hidden, created_at, updated_at) VALUES
(9540, 954, '列表', '', 2, 'POST /api/service/admin/recruitment/subscription/list', '', 'recruitment:push:list', 1, 1, 0, NOW(), NOW()),
(9541, 954, '测试', '', 2, 'POST /api/service/admin/recruitment/push/test', '', 'recruitment:push:test', 2, 1, 0, NOW(), NOW()),
(9542, 954, '保存', '', 2, 'POST /api/service/admin/recruitment/subscription/save', '', 'recruitment:push:save', 3, 1, 0, NOW(), NOW()),
(9543, 954, '删除', '', 2, 'POST /api/service/admin/recruitment/subscription/delete', '', 'recruitment:push:delete', 4, 1, 0, NOW(), NOW())
ON DUPLICATE KEY UPDATE parent_id=VALUES(parent_id), name=VALUES(name), icon=VALUES(icon), type=VALUES(type),
path=VALUES(path), component=VALUES(component), permission=VALUES(permission), sort=VALUES(sort),
status=VALUES(status), hidden=VALUES(hidden), deleted_at=NULL;
-- ============ 6) 超管角色绑定新菜单role_id=1 超管 ============
INSERT IGNORE INTO admin_role_menu (role_id, menu_id, created_at, updated_at)
SELECT 1, id, NOW(), NOW() FROM admin_menu
WHERE id IN (95,951,952,953,954,9510,9511,9520,9521,9530,9531,9540,9541,9542,9543);